Skip to content

feat(mdds)!: shard streaming pulls; size shards from request shape - #1215

Merged
userFRM merged 6 commits into
mainfrom
feat/generic-sharding
Jul 28, 2026
Merged

feat(mdds)!: shard streaming pulls; size shards from request shape#1215
userFRM merged 6 commits into
mainfrom
feat/generic-sharding

Conversation

@userFRM

@userFRM userFRM commented Jul 19, 2026

Copy link
Copy Markdown
Owner

Summary

Bulk-fetch sharding, made generic, resilient, and reachable from the streaming path. Two commits:

  1. The dedicated streaming builders (*_stream()) now shard concurrently like the buffered path — they previously always ran a single stream regardless of bulk_fetch or tier concurrency.
  2. Shard sizing no longer runs a density probe. plan_query decides shardability from the request shape and cuts the requested time/date range into equal concurrent bands. The OHLC sizing probe and its per-endpoint rows_per_event multiplier are deleted.
  3. A shard band failure no longer loses the whole pull.
  4. The two chunk-delivery implementations are unified onto one.

Sizing: the probe is gone

The old sizer fired a separate OHLC query to estimate a pull by counting trades and multiplying by a hardcoded per-endpoint constant (roughly 15 rows per trade for quote endpoints). Real NBBO density is two orders of magnitude higher, so large quote chains fell under the shard threshold and ran single-stream. Equal-time banding needs no estimate and no per-endpoint tuning; the concurrency win dominates the residual imbalance from equal-time versus equal-work cuts.

Resilience: a band failure bounds its blast radius

Previously one band's transport error cancelled the siblings and lost the entire pull.

  • Buffered (.await): a band's rows are held internally until the merge, so a failed band is re-fetched from scratch within the existing retry budget. Transient transport errors recover transparently, no duplicates.
  • Streaming (.stream): sibling bands drain to completion instead of being cancelled, and the returned Error::PartialShardFetch { failed } names the exact failed band windows, so a caller re-pulls only those slices instead of the whole result. A band that fails before delivering any chunk still retries transparently.

Delivery unification

The *_stream() builders used a codegen template that marked an empty keepalive chunk as "delivered," diverging from the macro path's deliver_chunk_slices on both retry budget and empty-query status. The template is deleted; both the sharded per-band arm and the single-stream arm now call deliver_chunk_slices, so the two public surfaces of an endpoint behave identically.

Generic and machine-enforced

No per-endpoint tuning remains. The shardable-endpoint set is now checked against the endpoint registry (intraday history = a history subcategory carrying start_time/end_time), so a newly added history endpoint cannot silently never-shard.

Breaking / API changes

  • bulk_fetch_plan is now fn(&self, endpoint, query) -> Option<ShardPlan> (was async fn -> Result<Option<ShardPlan>, Error>). It is pure computation on the request shape, so there is nothing to await and no error to surface. Rust-only; exposed in no binding.
  • New Error::PartialShardFetch variant (additive), mapped to StreamError / THETADATADX_ERR_STREAM in every binding beside the existing PartialReconnect.

Behavior notes (intentional)

  • A tick-interval pull now fans out on the window alone; a sparse one produces fast empty bands that fold to a single NotFound, so wall-clock is neutral.
  • A window under the 5-minute minimum band width stays on the single stream.
  • Merged output keeps the same canonical order; bulk_fetch = "off" bypasses sharding entirely.

Deadline

The 300s request timeout is a default. Large pulls should raise it via with_deadline / timeout_ms (Duration::ZERO disables it). Unchanged here.

Verification

  • cargo clippy --workspace --all-targets --locked -- -D warnings, cargo test --workspace + doctests, cargo fmt --check — clean
  • SDK-surface / docs-site / gRPC-snapshot regeneration — no drift; codegen deterministic
  • binding-parity, docs-consistency, lockfile-drift, and vocabulary gates — pass
  • New integration test (tests/sharded_fanout.rs, 7 cases): fan-out issues N band requests, merge/interleave order, NotFound folding, buffered re-fetch recovery, streaming partial-error naming the failed window
  • Two independent adversarial audits (band partition math; resilience/concurrency semantics) — 0 critical, 0 high

Note: check_c_abi_completeness flags 12 FFI config accessors (bulk_fetch / shard_concurrency / wait_mode / window sizes) declared in the header but unimplemented in thetadatadx-ffi/src. This is pre-existing on main (shipped in 0.2.0) and unrelated to this change.

userFRM added 6 commits July 19, 2026 12:20
The dedicated streaming builders (*_stream) now route through the bulk-fetch sharder like the buffered path; previously they always ran a single stream regardless of bulk_fetch policy or tier concurrency.

Shard sizing no longer runs a density probe. plan_query decides shardability from the request shape and cuts the requested time or date range into equal concurrent bands, removing the OHLC sizing probe and its per-endpoint rows_per_event multiplier; that multiplier mis-sized quote chains badly enough that large quote pulls never sharded.

BREAKING CHANGE: MarketDataClient::bulk_fetch_plan is now fn -> Option<ShardPlan> (was async fn -> Result<Option<ShardPlan>, Error>).
…eaming chunk delivery

A terminal band failure on a chunk-streaming sharded pull no longer cancels the sibling bands: they drain to completion, and the call returns the new Error::PartialShardFetch naming the failed band window(s) so the caller can re-pull exactly the missing slices instead of restarting the whole pull. A pull where no chunk reached the handler still fails wholesale with the underlying error, a band that fails before delivering anything still retries transparently, and the call deadline still cancels every band at once. The buffered path's per-band from-scratch re-fetch (attempt-local collection, dedup-free replay) is documented and pinned by tests. The new variant maps to StreamError / THETADATADX_ERR_STREAM across the Python, TypeScript, and C surfaces.

The four dedicated *_stream builders now drain through MarketDataClient::deliver_chunk_slices, the same primitive the generated .stream(handler) methods use, so both public surfaces of one endpoint agree on the no-replay delivered gating (marked only when rows reach the handler, keeping a recoverable transient after empty keepalives retryable) and stop at the first decode failure; the divergent for_each_chunk_body template is deleted.

The planner now logs at debug why a query did not shard (unlisted endpoint, no cut axis, width under two, malformed window, provably small grid, narrow window), and each band future runs inside a shard_band tracing span carrying its window plus a per-band completion line with rows and duration, so concurrent bands stay distinguishable in the log stream.

The shardable-endpoint set is now a const roster asserted equal to the registry's intraday-history endpoints (history* subcategory carrying start_time/end_time filters), so a new intraday history endpoint cannot silently never-shard. A new integration suite drives sharded pulls end-to-end through a per-band scripted mock, buffered and streaming: fan-out shape, band-order merge, NotFound folding, buffered band re-fetch, pre-delivery streaming retry, and the streaming partial-fetch error.
Whole-market as-of queries (stock/index at_time) return one row per day and are dominated by per-day server compute, not transfer. Banding the requested date range runs those per-day lookups concurrently over the tier's channel pool, so a wide as-of pull drops from a single stream to a fraction of its wall time (a one-year stock pull measured ~188s to ~18s at Pro width) with byte-identical, date-ordered output.

Add stock_at_time_trade, stock_at_time_quote and index_at_time_price to the shardable set, reusing the existing date-axis planner and ordered merge with no new axis. The set stays machine-tied to the registry: the tie now derives intraday history (history* with start_time/end_time) plus whole-market at_time (at_time subcategory, category not option). Single-contract option at_time carries a contract identity and is sparse over a wide range, so it stays single-stream.

The server computes at_time per day independently, with no last-tick carry across days (a pre-open time_of_day yields NotFound, not the prior close), so a date band seam cannot diverge from the single stream; verified against single-stream output value-for-value. Multi-day at_time .stream() now fans out, so its chunks arrive interleaved across bands rather than strictly date-ascending, matching every other sharded history stream.

Tests pin the plan (whole-market at_time to N date bands; option at_time and daily-only families decline) and the uncatchable residual (a concrete-contract tick over-shard is invisible to request shape).
An option-chain pull (strike wildcard) is a contract cross-product, and its cost is the server enumerating those contracts, not transfer. Slicing the requested window by time made every band re-enumerate the whole chain; splitting by right (call / put) instead makes each band assemble half the contracts. A both-rights chain now fans out into a call half and a put half, and the tier's remaining lanes slice each half by time (at Pro width 8, call x 4 time bands + put x 4), so the fan-out fills the tier's concurrency with work that divides the bottleneck rather than multiplying it.

The plan emits all-calls-then-all-puts, each half in time order. The ordered merge groups rows by (expiration, strike, right) and keeps band order within a contract, so it interleaves call before put within each strike and stays time-ascending — the exact single-stream canonical order, independent of the right x time split. Buffered output is therefore unchanged. A single-right chain (a strike wildcard already pinned to call or put) has nothing to divide on this axis and keeps the equal time split; a chain over a window too narrow for a time split still fans out into a plain call/put pair.

ShardBand::Time carries an optional right override applied per band through shard_apply_field, so both the buffered and streaming fan-outs pick it up. Tests pin the plan (both-rights chain -> call/put x time bands; single-right chain -> time-only; narrow concrete contract -> decline) and the merge (call/put x time bands reassemble into canonical contract order).
A `*` option chain at_time — every strike as-of a time, each day — is a dense per-day-compute pull that bands across a date range exactly like the whole-market stock and index at_time already do, but it was excluded along with the concrete single-contract case. Admit option_at_time_trade and option_at_time_quote to the shardable set and decline only the concrete contract (sparse, ~1 row/day) at runtime, so a chain shards and a single contract still runs on one stream.

Reuses the existing date-band and ChainKey merge that already serve multi-day history chains, so no new machinery is added. Date bands reproduce the single-stream row set, including at a carry-sensitive as-of time: an option's prior-day carry resolves from the calendar day before the pull, not from a band's start date, so band seams never diverge.
@userFRM
userFRM merged commit da44523 into main Jul 28, 2026
55 checks passed
@userFRM
userFRM deleted the feat/generic-sharding branch July 28, 2026 12:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant